Video processing apparatus and controlling method for same

ABSTRACT

A video processing apparatus that can be connected to a video playback apparatus includes an input unit configured to input video data from the video playback apparatus, an interpolation unit configured to generate interpolated frame image data of input video data, and an output unit configured to output the interpolated frame image data as interpolated video data, wherein the interpolation unit generates different interpolated frame image data according to a playback mode of the video playback apparatus.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/356,397, filed Jan. 20, 2009, which claims the benefit of JapaneseApplication No. 2008-021071, filed Jan. 31, 2008, both of which arehereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a video processing apparatus that canbe connected to a video playback apparatus, more particularly to a videoprocessing apparatus capable of generating different interpolated frameimages according to a playback mode of the video playback apparatus andperforming frame rate conversion.

2. Description of the Related Art

Recently, video equipment conducts frame rate conversion to display animage at a higher frame rate in order to suppress blur and flicker ofmoving images.

Japanese Patent Application Laid-Open No. 2001-054066 describes an imagedisplay system that differentiates the decoding process speed of amoving image read from a digital versatile disk (DVD) media between astandard playback mode and a special playback mode. Such processing canreduce the processing amount in slow playback, and can provide a smoothdisplay when fast-forward playback is performed.

Japanese Patent Application Laid-Open No. 03-263989 also describes atechnique of obtaining a motion vector from the preceding and subsequentframes, and using the motion vector to generate interpolation data as amethod for generating interpolated frames in frame rate conversionprocessing.

A frame rate conversion method in the conventional video processingapparatus, however, has a problem that a smooth image is not alwaysdisplayed when video data is input in the special playback mode. Thisproblem is in particular described as follows:

(1) In a Case of Fast-Forward Playback (FIG. 2)

When playback is carried out in the normal playback mode, frame imagesfrom input image data are continuously transferred into a frame rateconverter as images n0, n1, and n2. On the other hand, when thefast-forward playback is carried out, the frame images from the inputimage data are thinned to be images n0, n10, and n20 and transferredinto the frame rate converter. In this case, the frame rate convertergenerates an interpolated frame image n0′ from images n0 and n10, and aninterpolated frame image n10′ from images n10 and n20. When aninterpolated frame image is generated from a frame image temporarilyseparated as described above, an interpolation error can easily occurbecause of lower correlation between frames. When an image is generatedand displayed with the interpolation error, smoothness of a moving imageis impaired.

(2) In a Case of Frame-by-Frame Advance Playback (FIG. 3)

When a frame-by-frame advance playback is carried out, a frame image n0of an input video is input to a frame rate converter until the userstarts to perform the frame-by-frame advance playback. When the userstarts to perform the frame-by-frame advance playback (at the time ofthe user action in FIG. 3), the next frame image n1 is input. The framerate converter generates an interpolated frame image n0′ from frameimages n0 and n1 at a time when the frame is changed after the userperforms a playback action. Although the user wants to sequentially vieweach frame of the input video in frame-by-frame advance playback, theinterpolated frame image 0′ that does not originally exist is displayed.

(3) In a Case of Slow Playback (FIG. 4)

When a slow playback is carried out at a half speed, the same frame isinput to the frame rate converter successively twice such as n0, n0, n1,n1, and n2, n2. The frame rate converter generates an interpolated frameimage n0′ from frame images n0 and n1, and an interpolated frame imagen1′ from frame images n1 and n2.

Frame rate images output from the frame rate converter are images n0,n0, n0, n0′, n1, n1, n1, n1′, and n2, n2, n2. In other words, the sameimages alternatively appear a different number of times such as thrice(n0), once (n0′), thrice (n1), once (n1′), and thrice (n2). When timesof display of the same frame are different, the motion image isdisplayed with poor smoothness.

SUMMARY OF THE INVENTION

The present invention is directed to a video processing apparatus thatcan provide smooth display of a motion image by selectively generatingan interpolated frame image even when video data is input in a specialplayback mode.

According to an aspect of the present invention, a video processingapparatus that can be connected to a video playback apparatus includesan input unit configured to input video data from the video playbackapparatus, an interpolation unit configured to generate interpolatedframe image data of input video data, and an output unit configured tooutput video data interpolated with the interpolated frame image data,wherein the interpolation unit generates different interpolated frameimage data according to a playback mode of the video playback apparatus.

According to an exemplary embodiment of the present invention, the videoprocessing apparatus can conduct frame rate conversion so as to providea smooth motion image display even when video data is input in a specialplayback mode.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 is a block diagram of a video processing apparatus according toan exemplary embodiment of the present invention.

FIG. 2 is an explanatory diagram of the conventional technique in whichinput video data is reproduced in fast-forward playback.

FIG. 3 is an explanatory diagram of the conventional technique in whichinput video data is reproduced in frame-by-frame advance playback.

FIG. 4 is an explanatory diagram of the conventional technique in whichinput video data is reproduced in slow playback.

FIG. 5 is a block diagram of a correction unit according to an exemplaryembodiment of the present invention.

FIG. 6 is an explanatory diagram describing a display image of thepresent invention when an input video signal is reproduced in afast-forward playback.

FIG. 7 is an explanatory diagram describing a display image according tothe exemplary embodiment of the present invention when an input videosignal is reproduced in frame-by-frame advance playback.

FIG. 8 is an explanatory diagram describing a display image according tothe exemplary embodiment of the present invention when an input videosignal is reproduced in slow playback.

FIG. 9 is a block diagram of a video processing apparatus according toanother exemplary embodiment of the present invention.

FIG. 10 illustrates a flow chart describing an operation of a specialplayback detection unit illustrated in FIG. 9.

FIG. 11 illustrates a flow chart describing an operation of a specialplayback detection unit illustrated in FIG. 9.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

FIG. 1 is a block diagram of a video processing apparatus that can beconnected to a video playback apparatus according to an exemplaryembodiment of the present invention. Video data at a frame rate of 60 Hzis output from the video playback apparatus and input to an input unit101.

The video playback apparatus includes a video recorder, a video player,a digital camera, and a digital video camera.

An interpolation unit 102 conducts frame interpolation processing oninput video data, and outputs the input video data as video data at aframe rate of 120 Hz. An output unit 103 outputs the video data that isframe rate converted in a predetermined video format, to a displayapparatus.

A playback mode detection unit 104 receives a playback mode signal fromthe video playback apparatus and outputs playback mode information to acontrol unit 105. The playback mode information is information aboutplayback mode of the video playback apparatus selected by useroperation. The playback mode includes a normal playback mode and aspecial playback mode such as fast-forward playback mode, slow playbackmode, and frame-by-frame advance playback mode.

The interpolation unit 102 generates different interpolated frame imagesaccording to the playback mode information based on the control by acontrol unit 105 (MPU) as described below.

When the video playback apparatus is connected to the video processingapparatus via a digital interface such as a high-definition multimediainterface (HDMI), the video playback apparatus can be controlled by theoperation of the video processing apparatus. In other words, a playbackmode detection unit 104 detects a playback signal transmitted by a uservia a remote control of the video processing apparatus.

The detected playback mode information is notified to the video playbackapparatus via a consumer electronics control (CEC) line of the HDMI, anda playback unit of the video playback apparatus plays back a recordedvideo data based on the playback mode information.

FIG. 5 is a block diagram illustrating a structure of the interpolationunit 102.

A motion vector detection unit 402 obtains a motion vector from theframe n+1 and the frame n delayed in a frame delay unit 401. As a methodfor obtaining a motion vector, block-matching is often used which willbe described below. The frame n is separated into a block, eachconsisting of 16×16 pixels. Then, a search range of 48×48 pixels ismoved pixel by pixel from the same position of the search block on theframe n+1, and a position is sought where a sum of the absolutedifference is the smallest. The difference between the sought positionand the original block position becomes a motion vector.

The interpolation processing unit 403 generates an interpolated frameimage from the motion vector, an image of the frame n in the input videodata, and an image of the frame n+1. More specifically, each pixel valueof a block at the frame n and a block at the destination frame n+1 isaveraged and pasted at a position of one half of the motion vector onthe interpolated frame. In other words, the averaged frame image isgenerated using a plurality of frame images of the input video data.

A selection unit 404 selectively outputs video data output from theframe delay unit 401 and the averaged interpolated frame image datagenerated at the interpolation processing unit 403 according to theplayback mode information. In a case of the normal playback mode, theselection unit 404 outputs image data of double frame rate byalternatively outputting the input video data and the averagedinterpolated frame image data. In a case of the special playback mode,the averaged interpolated frame image data is not output but image dataof double frame rate is output by producing an output of the secondconsecutive input video data as the interpolated frame image data.

In other words, in case of the normal playback mode, the interpolationunit 102 uses a plurality of frame images of the input video data togenerate interpolated frame images that are different from the frameimage of the input video data. In a case of the special playback mode,the interpolation unit 102 generates the interpolated frame imagessimilar to the frame image of the input video data.

Linking the frame images displayed by the above operation is describedbelow for each of the special playback modes.

In a case of the fast-forward playback mode, frame rate conversion canbe performed without displaying the interpolated frame images n0′ andn10′ by issuing a preceding frame image to the position of theinterpolated frame as FIG. 6 illustrates.

In a case of the frame-by-frame advance playback mode, frame rateconversion can be performed by issuing a preceding frame image to theposition of the interpolated frame without displaying the interpolatedframe image n0′ that is absent in the output video data as FIG. 7illustrates.

In a case of the slow playback mode, the same frame image is evenlyrepeated four times by issuing a preceding frame to the position of theinterpolated frame as FIG. 8 illustrates. According to the presentexemplary embodiment, as described above, frame rate conversion isperformed by the motion vector interpolation when input video data isnot reproduced in special playback. On the other hand, the averagedinterpolated frame is not output, but input video data is output twicewhen input video data is reproduced in special playback. Thus, breakdownof the interpolated images at the time of fast-forward playback, displayof the interpolated frame at the time of frame-by-frame advanceplayback, and unnatural motions at the time of slow playback can beprevented.

According to the second exemplary embodiment of the present invention,the special playback detection unit detects the playback modeinformation without receiving the same from the outside.

FIG. 9 is a block diagram of the interpolation unit 102 according to thepresent exemplary embodiment.

The motion vector detection unit 902 detects motion vectors similar tothe first exemplary embodiment. The motion vector detection unit alsodetermines reliability of the motion vector. The reliability of themotion vector is determined by a sum of the absolute difference betweena block of the frame n and a block of the frame n+1. When the sum is aconstant value or higher, the unit determines that no reliability of themotion vectors is present because the block of the frame n and the blockof the frame n+1 are not similar to each other. When the sum is lowerthan the constant value, the unit determines the reliability to bepresent.

The special playback detection unit 905 detects whether the playbackmode is the special playback mode, considering the motion vectors andthe reliability of the motion vectors detected by the motion vectordetection unit 902. The operation of the special playback detection unit905 is described using flow charts illustrated in FIGS. 10 and 11.

FIG. 10 is a flow chart describing procedures for detecting thefast-forward playback mode among the special playback mode.

In step 1001, the special playback detection 905 counts block numbers ofthe motion vectors that shows no reliability within one frame. When thecounted value is less than the predetermined number (NO in step 1001),the processing proceeds to step 1004. In step 1004, the special playbackdetection unit determines the playback mode is not the fast-forwardplayback mode. When the counted value is a predetermined number or more(YES in step 1001), the processing proceeds to step 1002. In step 1002,the special playback detection unit 905 determines whether a state wherethe counted value is the predetermined number or more, continues for aperiod corresponding to the predetermined frame number or more. When thestate continues for a period corresponding to the predetermined framenumber or more (YES in step 1002), the processing proceeds to step 1003.In step 1003, the special playback detection unit 905 determines thatthe playback mode is the fast-forward playback mode. If the state doesnot continue for the period corresponding to the predetermined number ormore frames (NO in step 1002), the processing proceeds to step 1004. Instep 1004, the special playback detection unit 905 determines that theplayback mode is not the fast-forward playback mode.

Accordingly, the special playback detection unit 905 detects whether theplayback mode is the fast-forward playback mode.

Next, the procedures for detecting the frame-by-frame advance playbackmode and the slow payback mode are described using the flow chartillustrated in FIG. 11.

In step 1101, the special playback detection unit 905 detects whether astate where the motion vectors in the frame are all zero vectors (nomotion), continues for a period corresponding to the predetermined framenumber. If the state does not continue for the period corresponding tothe predetermined frame number (NO in step 1101), the processingproceeds to step 1105. In step 1105, the special playback detection unit905 determines that the playback mode is neither the frame-by-frameadvance playback nor the slow playback mode. When the state continuesfor the period of the predetermined frame number (YES in step 1101), theprocessing proceeds to step 1102.

In step 1102, the special playback detection unit 905 determines whetherthe motion vectors within one frame contain vectors other than a zerovector. When the motion vectors do not contain vectors other than a zerovector (i.e., all zero vectors) (NO in step 1102), the special playbackdetection unit 905 repeats step 1102 until the frames containing vectorsother than a zero vector are detected. When the motion vectors containvectors other than zero vectors (YES in step 1102), the processingproceeds to step 1103.

In step 1103, the special playback detection unit 905 determines whethera state where the motion vectors within one frame are all zero vectors(no motion), continues for a period corresponding to the predeterminedframe number. When the state does not continue for the period of thepredetermined frame number (NO in step 1103), the processing proceeds tostep 1105. In step 1105, the special playback detection unit 905determines that the playback mode is neither the frame-by-frame advanceplayback mode nor the slow playback mode. When the state continues forthe period of the predetermined frame number (YES in step 1103), theprocessing proceeds to step 1104. In step 1104, the special playbackdetection unit 905 determines that the playback mode is theframe-by-frame advance playback mode or the slow playback mode.

The special playback detection unit 905 decides that the playback modeinformation is the special playback mode when two results of the abovedetermination indicate the fast forward playback mode or theframe-by-frame advance playback mode/the slow playback mode. When theseresults do not indicate the fast forward playback mode and theframe-by-frame advance playback mode/the slow playback mode, the specialplayback detection unit 905 determines that the playback modeinformation is not the special playback mode.

The selection unit 404 determines the frame to be issued to the positionof the interpolated frame based on the playback mode information similarto the first exemplary embodiment.

According to the present exemplary embodiment as described above, themotion vectors are obtained from the video signals, and the playbackinformation is detected based on the motion vector information. Even ifno playback mode information is instructed from outside, frame rateconversion can be performed according to the playback mode.

Although the above embodiment describes the processing per frame unitfor the sake of simplification, the processing can also be provided perfield unit.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

What is claimed is:
 1. A video processing apparatus that can beconnected to a video source, the video processing apparatus comprising:an input unit configured to input video data from the video source; aninterpolation unit configured to generate interpolated frame image dataof input video data; and an output unit configured to output video datainterpolated with the interpolated frame image data, wherein theinterpolation unit generates the interpolated frame image data by usinga plurality of frame image data of the input video data when a playbackstate of the input video data is a normal playback state, and theinterpolation unit generates the interpolated frame image data that issubstantially the same as the frame image data of the input video datawhen the playback state of the input video data is a special playbackstate.
 2. The video processing apparatus according to claim 1, whereinthe special playback state includes at least one of fast-forwardplayback state, slow playback state, and frame-by-frame advance playbackstate.
 3. The video processing apparatus according to claim 1, whereinthe video source and the video processing apparatus are connected byusing a high-definition multimedia interface (HDMI), and whereinplayback state information indicating the playback state of the videodata is notified from the video source to the video processing apparatusvia consumer electronics control (CEC) lines of the HDMI.
 4. The videoprocessing apparatus according to claim 1, further comprising a motionvector detection unit configured to detect motion vectors relative tothe plurality of frame images of input video data, wherein theinterpolation unit generates the interpolated frame image data by usinga plurality of frame image data of input video data based on the motionvectors detected by the motion vector detection unit.
 5. The videoprocessing apparatus according to claim 4, wherein the motion vectordetection unit detects each motion vector for each of a plurality ofblocks in each of the plurality of frame images of the input video data,the video processing apparatus further comprising a determination unitconfigured to determine that the playback state of the video data is afast-forward playback state in a case where an equal or larger than apredetermined number of the frame images, in each of which are includedan equal or larger than a predetermined number of blocks each having amotion vector which is detected by the motion vector detection unit andof which reliability is lower than a standard level, continue.
 6. Thevideo processing apparatus according to claim 4, further comprising adetermination unit configured to determine that the playback state ofthe video data is either a slow playback state or a frame-by-frameadvance playback state in a case where an equal or larger than apredetermined number of the frame images, of which the motion vectorsdetected by the motion vector detection unit is zero, continue.
 7. Amethod for controlling a video processing apparatus that can beconnected to a video source, the method comprising: inputting video datafrom the video source; generating interpolated frame image data of inputvideo data; and outputting video data interpolated with the interpolatedframe image data, wherein the generating interpolated frame image dataof input video data comprises generating the interpolated frame imagedata by using a plurality of frame image data of the input video datawhen a playback state of the input video data is a normal playbackstate, and generating the interpolated frame image data that issubstantially the same as the frame image data of the input video datawhen the playback state of the input video data is a special playbackstate.
 8. The method according to claim 7, wherein the special playbackstate includes at least one of fast-forward playback state, slowplayback state, and frame-by-frame advance playback state.
 9. The methodaccording to claim 7, wherein the video source and the video processingapparatus are connected by using a high-definition multimedia interface(HDMI), and wherein playback state information indicating the playbackstate of the video data is notified from the video source to the videoprocessing apparatus via consumer electronics control (CEC) lines of theHDMI.
 10. The method according to claim 7, further comprising: detectingmotion vectors relative to the plurality of frame images of input videodata, wherein the generating interpolated frame image data comprisesgenerating the interpolated frame image data by using a plurality offrame image data of input video data based on the motion vectorsdetected in the detecting step.
 11. The method according to claim 10,wherein the detecting motion vectors comprises detecting each motionvector for each of a plurality of blocks in each of the plurality offrame images of the input video data, and further comprising:determining that the playback state of the video data is a fast-forwardplayback state in a case where an equal or larger than a predeterminednumber of the frame images, in each of which are included an equal orlarger than a predetermined number of blocks each having a motion vectorwhich is detected in the detecting step and of which reliability islower than a standard level, continue.
 12. The method according to claim10, further comprising: determining that the playback state of the videodata is either a slow playback state or a frame-by-frame advanceplayback state in a case where an equal or larger than a predeterminednumber of the frame images, of which the motion vectors detected in thedetecting step is zero, continue.
 13. The video processing apparatusaccording to claim 1, wherein the video source is a video playbackapparatus.
 14. The video processing apparatus according to claim 1,wherein the playback state is a playback mode.
 15. The method accordingto claim 7, wherein the video source is a video playback apparatus. 16.The method according to claim 7, wherein the playback state is aplayback mode.